Demo: Running a Neural Network Model and Tuning Its Hyperparameters (Self-Paced)

In this demonstration, you use a supervised machine learning algorithm to train a neural network a model. Using a pre-created flow, you also partition the data to perform honest assessment. Finally, you tune the hyperparameter values to improve the performance of the model.

Reminder: If you restarted your SAS session, or it has timed out due to inactivity, you must re-create the course libraries, LOCALLIB and VST. To do this, run the AssignLibrary flow.

1. In SAS Viya, launch SAS Studio by clicking the Applications menu button (the three-by-three grid in the upper left corner), expand ANALYTICS LIFE CYCLE, and click Develop Code and Flows.
2. In the navigation panel on the far left, click the Explorer button (second from top). Expand Files >Home > workshop > VST. Double-click AssignLibrary.flw to open it.
3. Click the Run button on the flow toolbar.

Demo Steps

1. Navigate to the VST folder and double click on Neural_Network_PVA_Final.flw.
2. Click on the SAS Program step and view the code. While the majority of the code is beyond the scope of this course, note the highlighted portions. The partition statement partitions the data into 50% for training and 50% for validation. The hidden statement creates 50 hidden units in the neural network. The REGL2=0.1 option in the OPTIMIZATION statement is used to apply an L2 (ridge) regularization. This adds a penalty to the sum of the squared slopes, which helps prevent overfitting by discouraging overly complex models. Here is the code:






proc nnet data=VST.PVA_DONORS_FINAL standardize=std; 
	partition fraction (validate=0.5 seed=12345); 
	target Response / level=nominal; 
	input GiftAvg36 PromCntAll PromCnt36 / level=interval; 
	input StatusCat96NK DemGender DemHomeOwner / level=nominal; 
	hidden 50; 
	train stagnation=5 outmodel=VST.Nnet_model; 
	optimization regL2=0.1; 
	*train stagnation=5 outmodel=VST.Nnet_model2; 
	*optimization regL2=0.001; 
run; 




In a practical scenario, you should consider all 12 features selected in the previous demo performing variable selection, but for the purposes of brevity, here we will be using only the top 6 features (inputs).

3. Click Run to execute the flow and open the results in a new browser tab.

The Model Information table includes information about the number of observations used to train the model, the number of hidden layers, hidden nodes, and the number of parameters estimated.



Note that the misclassification rate for validation data is 0.4511. Because the target variable is nominal, the algorithm tries to choose parameter estimates that can optimize the estimation criterion, which is the misclassification rate in this case.

The Iteration History table indicates that the algorithm performed 9 iterations in all, with the final model having the smallest validate error value (0.4511).

Note: Some SAS Visual Data Mining and Machine Learning models (including the neural network) are created using a nondeterministic process. This means that you might experience different results when you rerun the task.



To improve the performance of the model, you might want to add more inputs or tune the hyperparameter values, or both.

Next you will edit some of the hyperparameters.

4. Return to the flow and update the L2 hyperparameter from 0.1 to 0.001. To make this change, place an asterisk (*) before the first OPTIMIZATION statement and remove the asterisk from the second OPTIMIZATION statement. Do the same for the TRAIN statements: place an asterisk before the first statement and remove the asterisk from the second statement. This change will save the updated neural network as Nnet_model2 in the VST library. The code will now appear as follows:






proc nnet data=VST.PVA_DONORS_FINAL standardize=std;
	partition fraction (validate=0.5 seed=12345); target Response / level=nominal;
	input GiftAvg36 PromCntAll PromCnt36 / level=interval;
	input StatusCat96NK DemGender DemHomeOwner / level=nominal;
	hidden 50;
	*train stagnation=5 outmodel=Nnet_model;
	*optimization regL2=0.1;
	train stagnation=5 outmodel=VST.Nnet_model2;
	optimization regL2=0.001;
run;



5. Click Run to execute the flow and view the Results.

The Model information table indicates that the chosen model is the one with a misclassification rate on validation data equal to 0.4317, which is slightly lower than that of the previous model.

The Iteration History table indicates that the algorithm has performed as many as 103 iterations in the search for an optimal solution. However, the optimal solution was obtained at iteration number 53, producing the smallest misclassification rate on the validation data.



Note that the result produced by the neural network is not as interpretable as the regression models you saw earlier. This is often cited as one of the limitations of machine learning models, a topic that is discussed in the next section.